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1.  SUMMARY 


1.1  Background.  The  interoperability  test  methodology  investigation  was 
initiated  to  extend  existing  systems  testing  methodology  to  accomplish 
compatibility  and  interoperability  (C&I)  testing  of  Army  Battlefield  Automated 
Systems  (BAS)  at  the  Developmental  Test  (DT)  level.  The  current  phase  of  the 
investigation  included  the  development  of  a  Test  Operations  Procedure  (TOP) 
for  interoperability  testing. 

1.1.1  Problem.  DoD  has  developed  and  is  continuing  to  develop  a  number  of 
major  automated  command,  control,  communications  and  intelligence  (C3I) 
systems.  The  critical  element  in  the  success  or  failure  of  these  C3I  systems 
will  be  their  interoperability  and  performance  under  load  in  a  highly  interac¬ 
tive  tactical  environment.  To  date,  no  automated  comprehensive  test  capabil¬ 
ity  or  methodology  exists  which  can  fully  evaluate  the  interoperability  of 
these  complex  data  handling  systems,  much  less  groups  of  these  complex  sys¬ 
tems.  Only  independent  segments  of  key  performance  parameters  have  been  and 
can  be  evaluated  with  existing  simulation  and  field  type  test  facilities. 
This  deficiency  is  due  to  the  prohibitive  cost  of  assembling  all  the  interface 
hardware  systems,  troop  elements,  and  environment  generators  essential  to 
realistically  reproduce  realistically  the  loading  conditions  of  a  tactical 
environment.  As  a  result,  the  Army  is  developing  major  systems  with  a 
critical  role  in  the  combat  effectiveness  of  the  Field  Army  without  testing 
their  ability  to  perform  and  interoperate  under  anticipated  battlefield 
conditions. 

1.1.2  Progress.  Previous  phases  of  this  methodology  investigation  included 
the  following  activities: 

a.  Investigation  of  interoperability  testing  efforts  at  Army  and  other 
DoD  facilities. 

b.  Development  of  a  proposed  test  approach  and  measures  of  effectiveness 
(MOEs)  suitable  for  interoperability  testing. 

These  efforts  are  documented  in  the  fiscal  year  final  report,  dated  28  October 
1982. 


1.2  Objective.  The  objective  of  this  phase  of  the  investigation  was  to  adapt 
current  methodol ogi es  to  develop  guidelines  and  procedures  for  test  personnel 
to  conduct  C&I  testing  at  the  DT  level. 

1.3  Summary  of  Procedures.  The  methodology  investigation  analyzed  the 
principles  involved  in  C&I  testing  based  on  the  findings  of  previous  efforts 
and  related  projects.  Use  of  the  International  Standards  Organization  (ISO) 
Open  System  Interconnect  (OSI)  network  model  was  reevaluated  and  test 
instrumentation  suitable  for  C&I  testing  was  reviewed. 

1.4  Summary  of  Results. 

a.  A  test  methodology  was  proposed  which  consists  of  an  incremental  test 
approach,  a  network  model  (ISO  OSI),  and  MOEs  patterned  after  the  model.  The 
structured,  incremental  approach  provides  an  orderly  method  for  testing 
potentially  complex  networks  of  interoperating  systems  and  simulators/ 
stimulators.  Use  of  a  network  model  is  proposed  as  merely  an  aid  in 
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identifying  interoperability  functions  and  as  a  convenient  way  of  categorizing 
the  MOEs  and  test  parameters. 


b.  The  test  methodology  requires  test  instrumentation  capable  of  a 
distributed  configuration  of  modular  components  for  the  stimulation  and 
simulation  of  the  interoperable  network  of  systems.  The  Test  Item  Stimulator 
(TIS)  possesses  the  test  message  database,  real-time  simulation,  and  data 
reduction  and  analysis  capabilities  required  by  the  interoperability  test 
methodology.  Other  instrumentation  such  as  a  hybrid  monitor  (HM)  may  provide 
measurement  capabilities  suitable  for  some  aspects  of  C&I  testing. 

c.  An  interoperability  TOP  was  produced  to  describe  the  procedures  for 
performing  C&I  testing  of  systems. 

1.5  Analysis. 

a.  The  proposed  test  methodology  specifies  a  systematic  approach  to 
interoperability  testing.  Such  methods  could  increase  the  cost-effectiveness 
of  testing  by  isolating  problems  to  specific  component  system  functions  in  an 
efficient  manner. 

b.  TIS  functional  capabilities  and  its  flexible  configuration  provided 
by  a  modular  design,  satisfy  the  majority  of  the  instrumentation  needs  for 
interoperability  testing.  Further  enhancements,  such  as  integrating  hard¬ 
ware/software  monitors  into  the  TIS,  may  be  desirable  as  the  methodology 
evolves  with  practical  experience  on  tactical  systems.  Savings  in  interop¬ 
erability  test  resources  will  result  from  substituting  TISs  for  actual  BAS  in 
some  test  situations. 

c.  The  interoperability  TOP  is  a  first  attempt  at  defining  Interop¬ 
erability  test  procedures  at  the  DT  level.  Because  the  methodology  is  evolv¬ 
ing,  the  procedures  defined  in  the  working  draft  will  require  refinement  as 
they  are  applied. 

1.6  Conclusions. 


a.  An  approach  to  C&I  testing  has  been  proposed  which  would  use  test  in¬ 
strumentation  currently  being  developed  to  provide  cost-effective  test  proce¬ 
dures. 


b.  The  interoperability  TOP  developed  by  this  investigation  provides  a 
systems  approach  to  C&I  testing  which  will  identify  potential  interoperability 
problems  prior  to  system  fielding. 

1.7  Recommendations. 


a.  Functional  decomposition  of  interoperability  functions  using  a 
layered  network  model  should  be  validated  to  determine  its  applicability  to 
older  communication  schemes.  Actual  test  experience  should  be  used  for 
refining  and  generalizing  the  network  model  and  preliminary  MOEs. 

b.  The  interoperability  TOP  should  be  used  for  testing  a  tactical 
system.  Evaluation  and  refinement  of  the  procedures  should  be  performed 
during  this  validation  phase. 
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2.0  DETAILS  OF  INVESTIGATION.  The  definition  of  interoperability,  C&I  test 
responsibilities,  the  effect  of  current  interoperability  programs  on  U.S.  Army 
Test  and  Evaluation  Command  (TECOM)  activities,  trends  in  development  and 
testing  of  C3I  systems,  and  test  approaches  and  MOEs  used  by  various  test 
agencies  were  examined  in  prior  phases  of  the  investigation.  Results  of  prior 
phases  were  documented  in  the  fiscal  year  final  report,  dated  28  October  1982. 
The  final  phase  of  the  investigation  focused  on  developing  the  test 
methodology  using  the  TIS  test  instrumentation  architecture  and  an  interop¬ 
erability  model  for  identifying  system  under  test  (SUT)  interoperability 
functions.  A  TOP  for  interoperability  resulted  from  this  effort. 

2.1  Definition  of  Interoperability. 

a.  Interoperability,  as  applied  to  BAS,  is  the  ability  to  exchange  and 
process  information  between  (among)  systems.  Information  exchange  is  usually 
realized  through  the  use  of  digital  communication  channels  configured  into  a 
network.  Processing  of  the  exchanged  information  is  performed  through 
functions  of  the  SUT  which  may  be  termed  interoperability  functions. 

b.  The  interoperability  test  methodology  is  based  upon  function  tracing 
throughout  the  network  of  interoperable  systems.  Interoperability  functions 
perform  processes  which  may  be  categorized  as  information  (message)  exchange 
with  a  database,  SUT-specific  functions,  or  an  exchange  of  messages  with 
another  system.  Interoperability  of  the  network  of  systems  is  comprised  of 
compatibility  and  system  software  interoperability  issues. 

2.2  Major  Interoperability  Programs. 

a.  High  visibility  of  the  C&I  problem  at  all  levels  of  the  DOD  has 
resulted  in  the  creation  of  programs  and  assignment  of  responsibility  to  solve 
the  problem.  DoD  Directive  5000.3  outlines  Development  Test  and  Evaluation 
(DT&E),  specifically  including  C&I. 

b.  The  Army  Command  and  Control  Master  Plan  (ACZMP)  describes  a  systems 
approach  to  manage  interoperability  efforts  of  BAS.  The  Center  for  System 
Engineering  and  Integration  (CENSEI)  has  been  assigned  as  System  Engineer  for 
technical  execution  of  the  ACCS  and  has  developed  an  ACCS  Systems  Engineering 
Implementation  Plan  (SEIP).  This  plan  describes  the  systems  engineering 
functions  required  by  the  ACZMP  and  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC) -developed  Army  Battlefield  Interface  Concept  (ABIC). 
Interoperability  requirements  of  BAS  not  specifically  addressed  (n  the  AC2MP 
are  provided  by  the  ABIC.  The  test  activity  must  address  these  directives  and 
publications  to  define  interoperability  design  and  test  functions. 

c.  The  ACCS  program  extends  beyond  Army  BAS  to  include  coordination  with 
joint-service  and  international  programs.  CENSEI  has  responsibilities  in 
Joint  Interoperability  of  Tactical  Command  and  Control  Systems  (JINTACCS)  and 
NATO  programs  to  provide  maximum  flexibility  for  ACCS-developed  systems. 
Existing  standards  in  NATO  and  JINTACCS  documents  are  used  whenever  possible 
to  achieve  this  goal. 

2.2.1  Army  Command  and  Control  System. 

a.  The  ACCS  program  provides  the  systems  approach  required  for  compre¬ 
hensive  and  effective  C&I  testing.  Although  the  ACCS  itself  is  under  develop- 
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ment,  progress  to  date  has  resulted  in  a  well -organized  approach  to  the  system 
development  process.  The  top-down  management  approach  of  the  ACCS  program  is 
reflected  in  the  hierarchical  organization  of  the  ACCS  family  of  specifica¬ 
tions.  The  top-level  ACCS  specification  provides  the  technical  definition  of 
the  overall  ACCS.  This  document  introduces  the  concept  of  functional  and 
operational  subsystems  within  the  ACCS. 

b.  The  functional  area  subsystem  specification  corresponds  to  the  five 
functional  areas  ( Intel! igence/Electronic  Warfare  (EW),  Fire  Support,  Maneuver 
Control,  Air  Defense,  and  Combat  Service  Support)  of  the  TRADOC  Command  and 
Control  Subordinate  System  (CCS2)  architecture.  (Communications  comprises  a 
sixth  area  in  revised  versions  of  the  SEIP.)  Specifications  for  operational 
subsystems  correspond  to  the  basic  force  elements  of  Army  tactical  forces. 
These  include  five  division,  seven  separate  brigade,  and  three 
echelon-above-division  force  elements. 

c.  The  lowest  level  document  is  the  Interface  Specification  (IS).  The 
IS  provides  a  detailed  definition  of  requirements  to  be  satisfied  at  a  partic¬ 
ular  ACCS  component  system  interface.  The  IS  also  describes  the  inter¬ 
face/interoperability  standards  for  direct  and  indirect  interfaces.  Direct 
interfaces  involve  two  directly  coupled  ACCS  component  systems  while  indirect 
interfaces  imply  an  intervening  component  system(s).  External  interfaces  to 
joint  or  international  systems  provide  the  link  to  JINTACCS  and  NATO.  IS 
documents  include  physical,  electrical,  and  information  transfer  characteris¬ 
tics  of  the  interface. 


d.  The  test  philosophy  proposed  by  ACCS  includes  C&I  issues  as  a  funda¬ 
mental  aspect  of  DT.  C&I  testing  is  considered  a  part  of  the  normal  life- 
cycle  of  a  system  with  a  long-term  goal  of  interoperability  as  merely  another 
aspect  of  testing. 

2.2.2  Joint  Interoperability  of  Tactical  Command  and  Control  Systems. 


a.  Joint  Interoperability  Test  Force  (JITF)-JINTACCS  has  provided 
procedures  for  conducting  operational  C&I  testing.  Although  the  JINTACCS 
program  has  concentrated  on  testing  of  message  standards,  the  test  plans  and 
Technical  Interface  Design  Plans  (TIDP)  are  valuable  references  for  C&I 
testers.  While  specifically  oriented  toward  joint  systems,  much  of  the 
material  is  applicable  to  intraservice  testing.  The  operational  aspect  of 
JINTACCS  testing  is  appropriate  to  a  study  of  interoperability  at  the  DT  level 
because  of  increased  emphasis  on  combining  DT/Operational  Test  (OT)  as  a  means 
to  conserve  test  resources. 

b.  JINTACCS  also  places  component  systems  into  five  categories  or 
functional  segments,  similar  in  nature  to  the  ACCS  grouping.  This  categor¬ 
ization  aids  in  the  management  of  the  two  programs  and  does  not  necessarily 
reflect  a  difference  in  methodologies  required  to  test  C&I  issues  among  the 
various  system  types. 

2.3  Interoperability  Test  Methodology. 


a.  The  ACCS  and  JINTACCS  approaches  to  C&I  testing  are  valid  for  appli¬ 
cation  to  intraservice,  joint,  and  NATO-level  testing.  The  ACCS  systems 
approach  provides  a  hierarchy  of  specifications  from  total  system  to  component 
systems  and  provides  for  external  interfaces  to  satisfy  JINTACCS  and  NATO 
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requirements.  The  definition  of  functional /operational  subsystems  of  ACCS  and 
JINTACCS  is  a  management  aid  in  the  development  of  the  total  systems.  C&I 
testing  methodology  has  no  inherent  need  for  any  particular  classification 
scheme  except  where  logistics  or  system-specific  requirements  exist. 

b.  The  proposed  methodology  applies  an  incremental  approach  to  C&I 
testing.  The  methodology  uses  a  conceptual  model  of  an  interoperable  BAS  to 
aid  in  a  functional  decomposition  of  the  SUT.  A  network  model  (ISO  OSI)  is 
used  to  establish  hierarchical  procedures  and  MOEs  for  C&I  testing  of  the 
interfaces  between  C3I  systems.  The  structured  assessment  procedures  minimize 
the  test  effort  required  to  isolate  interoperability  problems  to  the  failing 
subsystem  component  and,  thus,  eliminate  expensive,  and  unnecessary,  testing 
of  higher  level  components  until  the  performance  of  supporting  lower  level 
components  has  been  verified. 

2.3.1  Incremental  Test  Method. 

a.  As  discussed  above,  the  ACCS  program  provides  a  systems  approach  that 
can  be  applied  to  C&I  testing.  Implied  in  the  ACCS  approach,  and  specifically 
mentioned  by  the  JITF,  is  the  concept  of  an  incremental  test  methodology. 
This  involves  the  test  of  a  component  system  to  verify  proper  operation  prior 
to  interoperability  testing.  If  this  test  is  successful,  further  testing 
occurs  with  other  interoperating  systems  added  until  the  interoperability  of 
the  total  system  is  assessed.  A  benefit  of  this  approach  is  the  ability  to 
discriminate  between  communication  and  BAS  malfunctions  with  minimum  test 
resources. 

b.  The  concepts  mentioned  above  were  refined  into  a  test  approach  for 
interoperability.  Test  design  should  be  performed  with  consideration  for  an 
incremental  ("building  block")  approach.  Scenario  requirements  should  allow 
for  single-thread  testing  to  demonstrate  proper  operation  of  the  SUT  and 
interoperable  systems  or  TISs  in  a  stand-alone  mode,  verify  the  interfaces  for 
compatibility,  and  ensure  that  test  control  and  communication  links  are 
functioning  properly.  This  phase  should  be  followed  by  multi-thread  testing 
of  C&I  issues.  The  SUT  and  test  instrumentation  configuration  should  follow  a 
similar  incremental  approach  with  components  colocated  initially,  evolving  to 
a  configuration  using  the  fully  extended  SUT  network  for  communication. 
Provision  should  be  made  for  trial  runs  using  a  "quick-look"  data  reduction 
and  analysis  capability  to  establish  performance  baselines  and  to  isolate 
problem  areas  as  soon  as  practicable.  The  eventual  SUT/test  instrumentation 
configuration  will  consist  of  interconnected  components  ( i . e . ,  the  SUT, 
available  interoperable  systems,  TISs  and  other  instrumentation)  to  form  a 
multiple  node  test  network  based  on  existing  and  planned  configurations  of  the 
SUT  network. 

2.3.2  Functional  Model.  The  interoperability  test  method  is  based  upon 
function  tracing  throughout  the  network  of  interoperable  systems.  To  reduce 
the  complexity  of  testing,  compatibility  issues  were  isolated  from  higher 
level  interoperability  functions.  A  conceptual  model  of  interoperability 
processes  was  then  formed  to  simplify  (message)  exchange  with  a  database,  SUT- 
specific  functions,  or  an  exchange  of  messages  with  another  interoperating 
system.  Further  explanation  of  the  functional  model  is  provided  in  appendix  D 
of  the  TOP.  The  interoperability  test  planning  portion  of  the  TOP  describes 
the  procedure  for  producing  a  functional  requirements  matrix  based  on 
interoperability  processes  of  the  SUT. 


2.3.3  Network  Model .  The  ISO  OSI  reference  model  was  chosen  as  a  basis  for 
organizing  test  aspects  of  communications  interfaces.  The  model  (see  figure 
1)  may  be  used  to  separate  C&I  processes  as  previously  mentioned.  Utility  is 
also  found  in  defining  test  instrumentation  requirements,  structuring  tests 
(from  lower  to  higher  levels)  for  incremental  testing,  as  an  aid  in 
identifying  interoperability  functions,  and  as  a  means  of  categorizing  MOEs 
and  test  parameters.  Details  are  provided  in  the  previously  referenced  fiscal 
year  report. 

2.3.4  Measures  of  Effectiveness. 

a.  Preliminary  MOEs  were  proposed  in  an  earlier  phase  of  the  inves¬ 
tigation.  For  purposes  of  DT  level  interoperability  testing,  MOE  and  measure 
of  performance  (MOP)  are  considered  synonymous. 

b.  The  layered  network  model  can  provide  the  foundation  for  the  test 
criteria  and  parameters  that  pertain  to  interoperability  issues.  The  inter¬ 
face  that  is  being  tested  can  be  subdivided  into  the  seven,  more  manageable 
layers  (or  levels)  of  the  ISO  OSI  model.  Test  parameters  may  then  be  iden¬ 
tified  for  each  level  of  the  interface  and  combined  to  produce  a  hierarchy  of 
MOEs. 

c.  Some  MOEs  associated  with  the  lower  layers  (compatibility  issues)  of 
an  interface  are  dependent  upon  the  physical  channel  characteristics,  while 
MOEs  for  higher  layers  reflect  SUT-specific  functions.  Structured  design 
techniques  being  used  on  newer  systems  (e.g..  Position  Location  Reporting 
System  (PLRS)/Joint  Tactical  Information  Distribution  System  (JTIDS)  Hybrid 
(PJH)  use  of  X.25  protocol)  may  allow  generic  MOEs  to  be  developed  in  the 
future. 

2.4  Interoperability  Test  Instrumentation. 

a.  Existing  methodologies  and  test  instrumentation  used  for  single 
system  testing  were  reevaluated  in  terms  of  interoperability  test  demands. 
This  process  produced  the  requirements  for  test  instrumentation  that  provides 
for  flexible  configuration  and  compatibility  with  existing  and  planned  tac¬ 
tical  systems.  A  separate  project  evolved  from  this  effort  and  is  currently 
developing  the  TIS  to  satisfy  these  initial  interoperability  instrumentation 
requirements. 

b.  Other  test  drivers,  similar  conceptually  to  TIS,  were  examined  as 
well  as  other  instrumentation  which  would  enhance  the  measurement  capability 
for  interoperability  testing.  This  instrumentation  is  briefly  described 
below. 


2.4.1  Test  Item  Stimulator.  The  Interim  Test  Item  Stimulator  (ITIS)  was  a 
test  driver  which  was  developed  by  the  U.S.  Army  Electronic  Proving  Ground 
(USAEPG)  for  DT  of  the  Maneuver  Control  System  (MCS).  The  ITIS  proved  useful 
for  single-system  testing  with  prescripted  message  streams.  Interoperability 
testing,  by  definition,  requires  the  ability  to  simultaneously  test  multiple 
systems  and  allow  for  the  real-time  generation  and  insertion  of  messages  into 
test  message  streams.  Thus,  the  ITIS  has  evolved  into  the  TIS  to  meet  these 
additional  test  requirements.  Test  conduct  using  the  TIS  is  separated 
functionally  into  three  phases:  pre-test  scenario  preparation,  real-time  item 


Application 


Application  protocol 


Name  of  unit 
exchanged 


Application 


Presentation 


Presentation  protocol 


Presentation 


Session  protocol 


Transport 


Transport  protocol 


Communication  subnet  boundary 


Transport 


Physical 


Physical 


Network 


Internal  subnet  protocol 


Oata  link 


Physical 


Data  link 


Physical 


1 —  Network  layer  host  -  IMP  protocol 

—  Oata  link  layer  host  -  IMP  protocol 

—  Physical  layer  host  -  IMP  protocol  ^ 

TITLE 

DESCRIPTION 

Physical  Layer 

Physical  connections  necessary  to  transmit  data 
on  a  bit  I/O  level 

Data  Link  Layer 

Transforms  raw  bits  into  error-free  line  to 
network  layer 

Network  Layer 

Groups  data  into  packets,  routes  packets  to 
destination,  performs  error  accounting 

Transport  Layer 

Accepts  data  from  session  layer,  forwards  to 
network  layer,  assures  end-to-end  accountability 

Session  Layer 

User  interface  to  network,  handles  connection 
establishment 

Presentation  Layer 

Library  of  common  application  functions  shared 
among  users 

Application  Layer 

Unique  messages  handling  specific  to  application 

Figure  7. 
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stimulator,  and  post-test  data  reduction  and  analysis.  The  TIS  design  is 
described  in  more  detail  in  appendix  C. 

2.4.2  Air  Force  Simulation,  Monitoring,  Analysis,  Reduction,  and  Test  System. 
Test  instrumentation  to  complement  the  procedures  of  JINTACCS  is  being 
developed  by  the  Air  Force  Simulation,  Monitoring,  Analysis,  Reduction,  and 
Test  System  (SMARTS).  The  proposed  SMARTS  would  automate  many  of  the  process¬ 
es  currently  performed  manually.  Like  the  JITF,  SMARTS  allows  centralized 
control  of  decentralized  test  facilities,  independent  testing  by  the  decen¬ 
tralized  test  facilities,  and  flexible  test  configuration  and  will  have  the 
capability  to  support  intraservice,  joint,  and  NATO-level  testing. 

2.4.3  Hybrid  Monitor. 

a.  The  HM  concept  (an  integration  of  both  hardware  and  software  monitors 
into  a  central  monitor  system)  evolved  from  earlier  investigations  of  stand¬ 
alone  hardware  and  software  monitors.  The  concept  of  integrating  the  capabil¬ 
ities  of  both  hardware  and  software  monitors  controlled  by  a  central  monitor  - 
the  HM  approach  -  was  developed  in  an  attempt  to  exploit  the  desirable  fea¬ 
tures  of  both  monitors  while  minimizing  their  shortcomings. 

b.  The  multiprocessor  hybrid  performance  monitor  potentially  has  several 
advantages  over  hardware  or  software  monitors.  It  is  able  to  monitor  a 
network  of  processors  simultaneously.  The  hybrid  should  also  prove  to  be 
easily  reconfigurable  and  allow  the  tester  to  approach  collection  of  data 
using  several  methods  which  blend  the  degree  of  hardware  and  software 
monitoring  required  for  a  particular  system. 

2.5  Technological  and  Economic  Constraints. 

a.  The  nature  of  interoperability  necessitates  the  involvement  of 
multiple  systems.  The  number  of  systems  involved  forms  the  basis  for  techni¬ 
cal  and  economic  concerns  of  considerable  magnitude.  Technical  problems  for 
C&I  testing  parallel  those  encountered  in  developing  the  sophisticated  C3I 
systems  themselves.  The  high  data  rates  and  large  volume  of  data  exchange 
place  large  demands  on  the  measurement  system  architecture  and  data  reduction 
and  analysis  capabilities.  Limited  resources  often  prevent  the  assembly  of 
the  necessary  interoperating  systems  and  personnel  and  place  even  greater 
technical  demands  on  the  test  activity.  Development  of  stimulators/ 
simulators,  such  as  the  TIS,  is  a  cost-effective  way  to  minimize  the  actual 
tactical  system  equipments  required  during  interoperability  testing. 

b.  The  large  volumes  of  data  inherent  in  interoperability  testing  may  be 
reduced  through  the  use  of  special  techniques  such  as  exception  reporting  and 
statistical  data  recording,  or  recording  only  samples  of  the  raw  data.  Test 
instrumentation  must  possess  automated  data  reduction  and  analysis  capabil¬ 
ities  to  allow  effective  analysis  of  the  test  data. 

c.  The  effect  of  limited  economic  resources  upon  comprehensive  C&I 
testing  may  be  reduced  by  combining  DT/OT  tests--combined  DT/OT  should  use 
common  test  issues,  the  same  instrumentation,  and  the  same  philosophy  of 
following  a  scenario  script  whenever  practicable.  The  ACCS  program  endorses 
such  testing  to  conserve  test  resources. 
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2.6  Interoperability  Test  Operations  Procedure.  An  interoperability  TOP  was 
developed  to  describe  the  facilities,  instrumentation,  preparation  for  tests, 
test  categories,  test  planning,  test  controls,  test  methods,  data  reduction 
and  presentation  methods. 


EXHIBIT  P-16  (Part  I)  Date:  1983 

Production  Engineering  Measure  (PEM)  Project 
RCS  DRCMT-835 

SUBTASK  No.  67 


I.  Project  Number:  0855071  (TECOM)  2.  Fiscal  Code:  PA  5397  3.  Cost:  $100K 

4.  Project  Title:  Interoperability  Test  Methodology 

5.  Name  and  location  of  facility/contractor:  US  Army  Electronic  Proving 
Ground,  Fort  Huachuca,  AZ  85613 

6.  General  Objective:  Reduced  costs;  increase  efficiency. 

7.  Problem:  Only  independent  segments  of  key  performance  parameters  have 
been  and  can  be  evaluated  with  existing  simulation  and  field  test  facilities. 
The  deficiency  is  due  to  prohibitive  costs  of  assembling  all  the  interfacing 
hardware  systems,  troop  elements,  and  environment  generators  essential  to 
realistically  reproduce  the  loading  conditions  of  a  tactical  environment.  As 
a  result,  the  Army  will  produce  major  systems  with  a  critical  role  in  the 
combat  effectiveness  of  the  Field  Army  without  adequately  testing  their 
ability  to  perform  and  interoperate  under  anticipated  battlefield  conditions. 
Current  methodologies  must  be  upgraded  to  fully  evaluate  the  interoperability 
of  production  automated  systems. 

8.  Proposed  Solution:  Automate  state-of-the-art  methodologies  and  establish 
appropriate  simulations,  including  computer  simulations  and  physical  simula¬ 
tors;  or  drivers  to  simulate  interactive  interoperants  interfacing  with  the 
test  systems.  General  purpose  concepts  will  be  emphasized  to  make  the  ca¬ 
pability  applicable  to  the  broad  spectrum  of  modern  battlefield  electronic 
systems  which  will  require  performance  validation  during  production  testing. 

9.  Justification:  If  this  task  is  not  accomplished,  it  will  be  infeasible  to 
conduct  adequate  interoperability  testing  on  a  broad  spectrum  of  major, 
electronically  based,  computer  driven  materiel  systems.  As  a  result,  there 
will  be  a  significant  lack  of  assurance  that  these  systems  will  interoperate 
with  the  many  other  interoperants  on  the  battlefield.  The  risk  accompanying 
such  a  situation  would  be  intolerably  high. 

10.  Benefits: 

a.  Quantifiable  benefits  (S/I):  None  Basis: 

b.  Non-quantifiable  benefits:  Dollar  savings  cannot  be  quantified  since 
there  is  no  economically  feasible  alternative  which  would  accomplish  the 
requirement  for  interoperability  validation. 

II.  Deliverables:  A  description  of  the  techniques  employed,  the  results, 
specifications,  limitations  of  the  defined  production  test  capability  and 
other  pertinent  information  will  be  documented  in  a  final  report  at  the 
conclusion  of  this  effort. 
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12.  Funding  Profile  and  Scheduled  Technical  Completion  Dates: 


Fiscal  Year  $Costs,  (XK)  Month-Year 

FY  85  $100K  Sep  85 

FY  M  $100K  Sep  86 

FY 

TOTAL 

13.  End  Items  Supported:  The  following  generic  types  of  production  systems 

will  be  supported. 

a.  Primary  -  Intelligence  Systems;  Electronic  Warfare  Systems;  Command 

and  Control  Systems  and  Communications/Navigation  Systems 

b.  Secondary  -  N/A. 

14.  Key  Milestone  Dates: 

a.  PEP  Completion  for  primary  end  item  -  N/A 

b.  MMT  Completion  -  September  1986 

c.  Primary  End  Item  TC  -  N/A 

d.  Start  of  Full  Scale  Production  -  N/A 

e.  Preliminary  Design  Criteria  for  Facility  -  N/A 

15.  Related  MMT  and  Feasibility  Demonstration  Efforts: 

a.  Project  Nos.  67 _  _  _  _ 

b.  Initiation  Date  Jan  80  _  _  _ 

c.  Completion  Date  Sep  85  _  ’ _ __J 

16.  Plan  for  Implementation  of  Efforts'  Results: 

a.  When  -  FY85/86 

b.  Where  -  US  Army  Electronic  Proving  Ground 

c.  How  -  Procure  the  required  test  equipment  and  facilities. 

d.  Who  -  TECOM  activities  responsible  for  interoperability  testing  of 

automated  systems. 

e.  Cost  -  Cost  of  implementation  will  be  dependent  upon  the  required 

equipment/facilities  defined  by  this  task,  and  thus  cannot  be 
quantified  at  this  time. 

17.  Energy  Resource  Impact  Statement:  Study  does  not  require  resources 

beyond  those  required  for  study,  analysis,  and  computer  program 
generation;  therefore,  no  impact  is  expected  on  energy  re¬ 
sources. 

Project  Engineer: 

a.  Name  -  Larry  W.  Miller 

b.  Organization  -  Cdr,  US  Army  Test  &  Evaluation  Command,  ATTN: 

AMSTE-AD-M,  APG,  MD  21005-5055 

c.  Phone  Numbers,  AUT0V0N  283-3677  Commercial  301-278-3677 

Enclosure  1  -  Environmental  Documentation 
None  required. 


ABIC 


Army  Battlefield  Interface  Concept 

ACCS  .  Army  Command  and  Control  Systems 

AC2MP  .  US  Army  Command  and  Control  Master  Plan 

BAS  .  Battlefield  Automated  Systems 

CCS  .  Communi cations  Control  System  (Advanced  Field  Artillery 

Tactical  Data  Systems) 

CCS2  .  Command  and  Control  Subordinate  System 

CENSEI  .  Center  for  System  Engineering  and  Integration 

C&I  .  Compatibility  and  Interoperability 

C3I  .  Command,  Control,  Commurr cations ,  and  Intelligence 

CTP  .  Coordinated  Test  Plans 

DBMS  .  Data  Base  Management  System 

DT .  Developmental  Test 

DT&E  .  Developmental  Test  and  Evaluation 

EDC  .  Error  Detection  and  Correction 

EFL  .  Event  Format  Library 

EPUU  .  Enhanced  PLRS  User  Unit 

EW  .  Electronic  Warfare 

HIU  .  Host  Interface  Unit 

HM  .  Hybrid  Monitor 

IS  .  Interface  Specification 

ISO  .  International  Standards  Organization 

ITIS  .  Interim  Test  Item  Stimulator 

JINTACCS  .  Joint  Interoperability  of  Tactical  Command  and  Control  Systems 

JITF  .  Joint  Interoperability  Test  Force 

JTIDS  .  Joint  Tactical  Information  Distribution  System 

MCS  .  Maneuver  Control  System 

MFL  .  Message  Format  Library 


MOE  .  Measure  of  Effectiveness 

MOP  .  Measure  of  Performance 

OSI  .  Open  System  Interconnect 

OT  .  Operational  Test 

PEM  .  Production  Engineers  Measure 

PLRS  .  Position  Location  Reporting  System 

PJH  .  PLRS/ JT I DS  Hybrid 

SEIP  .  Systems  Engineering  Implementation  Plan 

SMARTS  .  Air  Force  Simulation,  Monitoring,  Analysis,  Reduction,  and  Test 

System 

SSA  .  System-Specific  Applique 

SUT  .  System  Under  Test 

TCT  .  Tactical  Computer  Terminal 

TDC  .  Time  Dispersal  Coding 

TECOM .  U.S.  Army  and  Evaluation  Command 

TIDP  .  Technical  Interface  Design  Plans 

TIS  .  Test  Item  Stimulator 

TOP  .  Test  Operations  Procedure 

TRADOC  .  U.S.  Army  Training  and  Doctrine  Command 

URO  .  User  Read-Out 

USAEPG  .  U.S.  Army  Electronic  Proving  Ground 


18 


1.  Test  Item  Stimulator. 

a.  The  TIS  has  evolved  from  the  ITIS  in  order  to  meet  the  additional 
requirements  of  interoperability  testing.  The  TIS  was  designed  to  provide  the 
capability  to  generate  message  traffic  to  support  the  testing  of  C3I  systems 
including: 

.  JTIDS  class  2  terminal  using  TADIL-J  messages. 

.  TSQ-73  host  interface  unit  (HIU)  using  ATDL-1  and  TADIL-B  messages. 

.  Hawk  HIU  using  ATDL-1  messages. 

.  Tactical  computer  terminal  (TCT)  HIU  using  MCS  messages. 

.  TACFIRE  Communications  Control  System  (CCS)  using  TACFIRE  CCS  mes¬ 
sages. 

.  PJH  enhanced  PLRS  using  unit  (EPUU)  using  user  read-out  (URO)  message 
and  EPUU  messages. 

b.  The  pre-test,  real-time,  and  post-test  functions  run  on  the  same 
VAX  11-based  system.  As  a  result,  all  three  phases  have  access  to  the  test 
message  database.  The  pre-test  software  provides  the  interactive  capability 
to  specify  test  conditions  and  maintain  a  library  of  message  formats  (MFL), 
and  generate  scenario  events.  Scenario  messages  are  organized  into  events 
which,  together  with  the  underlying  format  definitions,  comprise  the  Event 
Format  Library  (EFL).  The  real-time  software  generates,  monitors,  and  records 
message  traffic  during  the  tests,  providing  for  the  interactive  control  of  the 
test  environment  and  parameters.  The  design  also  includes  the  capability  for 
real-time  operator-oriented  event  generation.  The  post-test  software  process¬ 
es  test  data  for  test  report  generation. 

2.  Pre-Test.  The  pre-test  function  supports  on-line  generation,  review,  and 
modification  of  test  scenarios.  The  test  scenarios  consist  of  time-sequenced 
events  and  test  actions  which  change  the  value  of  a  system  simulation 
parameter,  provide  test  control  information,  or  trigger  required  responses. 
Two  types  of  events  occur  in  scenarios: 

a.  Prescripted  events  -  generated  from  pre-test  operator-entered  mes¬ 
sages  and  directly  transmitted  to  the  SUT  during  real-time  processing. 

b.  Real-time  events  -  generated  from  pre-test  operator-entered  events 
that  are  manipulated  during  real-time  testing  to  generate  messages  for  trans¬ 
mission  to  the  SUT. 

3.  Real-Time. 

a.  The  real-time  function  provides  the  interface  for  stimulating  and 
monitoring  the  SUT.  The  real-time  function  processes  both  scenario  and 
operator-entered  messages,  producing  a  message  stream  to  stimulate  the  SUT. 
The  resulting  message  exchange  is  logged  for  later  processing.  The  protocol 
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handlers,  formatters,  and  interface  elements  dealing  with  a  specific  protocol 
are  collectively  referred  to  as  a  system-specific  applique  (SSA). 


b.  Scenario-based  and  operator-entered  messages  driving  the  real-time 
processing  are  scheduled  through  the  event  reader.  Command  messages  are 
routed  to  the  SSA  control  process,  where  they  modify  the  test  execution. 
Pre-scripted  messages  are  sent  directly  to  the  transmit  process,  which  trans¬ 
mits  data  to  the  SUT  and  logs  the  transmission.  Real-time  messages  and 
response  messages  are  routed  to  the  real-time  message  generation  and  response 
handling  processes,  which  in  turn,  send  transmittable  messages  to  the  transmit 
process.  The  receive  process  logs  the  SUT  messages  received  and  sends  the 
received  data  to  the  response  handling  task,  possibly  triggering  a  response. 
Test  notes  are  displayed  to  the  operator  and  routed  to  the  logging  process. 

4.  Post-Test.  Post-test  processing  of  the  log  files  generated  during  testing 
produces  statistical  reports  on  message  content  and  end-to-end  system 
throughput. 

5.  Test  Configuration  Data. 

a.  The  TIS  is  designed  to  operate  in  a  network  configuration  with  other 
TISs  during  a  test. 

b.  Each  TIS  consists  of  four  SSAs  which  are  driven  by  separate  scenarios 
and  which  are  capable  of  stimulating  different  types  of  systems.  The 
configuration  of  one  TIS  is  described  through  the  use  of  Test  Description  and 
Test  Composition  information  for  each  SSA.  A  System  ID  is  used  to  reference 
various  records  describing  scenarios  and  formats.  This  convention  allows  for 
the  independent  naming  of  entities  relating  to  a  new  system  and  provides 
multiple  system  support. 

6.  Use  of  the  International  Standards  Organization  Open  System  Interconnect 
Reference  Model . 

a.  It  is  apparent  that  support  of  such  widely  diverse  systems  with  a 
single  TIS  requires  a  well -coordinated  design  philosophy.  It  is  necessary, 
therefore,  to  establish  a  common  base  for  comparison  of  tactical  communication 
protocols.  The  ISO  reference  model's  layered  concept  of  a  communications 
protocol  provides  this  basis. 

b.  Communication  with  each  system  requires  some  rudimentary  communica¬ 
tion  protocol  interface.  In  the  TIS,  this  type  of  protocol  handling  is 
performed  in  the  SSA.  The  SSA  is  the  part  of  the  TIS  real-time  software  that 
performs  highly  specialized  functions  requiring  re-implementation  from 
SUT-to-SUT. 

c.  Layers  1,  2,  and  3  of  the  ISO  model  are  necessary  for  any  message 
exchange  to  occur.  These  layers  are  mandatory  for  minimal  SSA  implementation. 
Layer  4  is  a  bridge  between  the  essential  lower  three  layers  and  the  sys¬ 
tem-tailored  upper  three  layers.  Layer  4  assures  end-to-end  message  transfer 
and  provides  logical  (named)  rather  than  physical  (hard-wired)  addressing  of 
nodes.  It  is  highly  desirable  to  implement  the  layer  4  function  in  an  SSA. 
This  allows  logical  node  addressing  on  the  message-generation  level.  Process¬ 
es  representing  layers  5,  6,  and  7  are  not  essential  to  message  exchange. 
Omission  of  processes  representing  layers  5  to  7  may  cause  error  conditions  or 


Table  I.  MCS  and  TACFIRE  Processes  Mapped  to  ISO  Model 


- m - 

- ISO  MOOR - 

- TACFIRE - 

Layer  7 

Tactical  Event  Simulation 

Message  Format  Definition 

Application  Layer 

Message  Format  Definition 

Layer  6 

SYS; FORM  Format  Skeleton 
Transmission 

Abridging  of  Messages 

Presentation  Layer 

Message  Compaction 

Remote  Requests  (filing, 
deletion,  retrieval) 

Layer  5 

Transport  Layer 

Serialization,  Validation 

Layer  4 

Message  of  Interest  Routing 

End-to-End  Accountability 

Transport  Layer 

Remote  Loop  Test 

Routing/ Re lay 

Layer  3 

Autodial 

Network  Layer 

Subscriber  Table 

EOC/TDC/Double  Blocking 

Layer  2 

EDC/TDC 

ACK/AUTORETRY 

Data  Link  Layer 

ACK/NAK/AUTORETRY 

FSK  4-Wire  600,  1200  Baud 

Conditioned  Di phase 

8K,  16K,  32K  Baud 

Layer  1 

Physical  Layer 

FSK  4-Wire  600,  1200  Baud 

55-Wire  Parallel  Interface 

TEST  SYSTEM 

CONTROL  - ►  - -  SPECIFIC 
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ATTN:  STEEP-TM-AC  4 

Fort  Huachuca,  AZ  85613-7110 


Addressee 


Commander 

US  Army  Tropic  Test  Center 
ATTN:  STETC-TD-AB 
APO  Miami  34004-5000 

Commander 

US  Army  White  Sands  Missile  Range 
ATTN:  STEWS-TE-PY 
STEWS-TE-0 
STEWS-TE-M 
STEWS-TE-A 

White  Sands  Missile  Range,  NM.  88002-5000 


